Phone-Level Prosody Modelling With GMM-Based MDN for Diverse and Controllable Speech Synthesis

Generating natural speech with a diverse and smooth prosody pattern is challenging task. Although random sampling phone-level distribution has been investigated to generate different patterns, the diversity of generated still very limited far from what can be achieved by humans. This largely due use uni-modal distribution, such as single Gaussian, in prior works modelling. In this work, we propose novel approach that models prosodies GMM-based mixture density network(MDN) then extend it for multi-speaker TTS using speaker adaptation transforms Gaussian means variances. Furthermore, show clone reference components produce prosodies. Our experiments on LJSpeech LibriTTS dataset proposed method MDN not only achieves significantly better than both single-speaker TTS, but also provides naturalness. The cloning demonstrate similarity comparable recent fine-grained VAE while target better.